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^ SUMMARY 
Problem 

'V — 

AltJhough the individualized instruction provided by the Navy*s computer- managed 
instruction (CMI) system allows students to progress according to their ability, a strong 
motivating mechanism is needed to ensure that students meet predicted course comple- 
tion schedules. Any innovation that improves student progress rates has the potential to 
reduce training time and costs. Various incentives to improve achievement have been 
introduce in Navy training schools, but fully effective methods for motivating students in 
CMI courses are ^ill needed. 

Obiectives 

The objectives of this research were to (1) develop feedback charts that dispUy 
informatioa on cumoilative progress, to motivate students, (2) determine the be^ pro- 
cedures for chart delivery, arx} (3) assess the feasibility .of the chart pr6cedures in 
operational CMI courses. 

Approach 

The approach for this investigation^ipvolyed conducting two experiments. Experi- 
ment I was designed to test five types of chart conditions in one CMI course^ Experijy>€nt 
n was an operational test of the best chart procedure .from Experiment I in four CMI 
schools with varying' content and management styles. 

Results . ^ . t 

Experiment I yielded no significant differences between t-he five chart procedures in^ 
.terms^ actual course completion times. The best chart method appeared to be the 
.condition in which students requested charts, that contained indications of available 
incentives. Student and instructor attitudes toward the chart procedures were'strongly 
supportive. Students rated the desirability of several possible incentive options, ^A 
special service ribbon for academic accomplishment was highly cated by students as an 
incentive option/ ' ' , 

Experiment II was conducted in operational CMI learning complexes in the following 
four schools: Basic Electricity and Electronics, Great Lakes and Memphis, Propulsion 
Engineering Basics, and the Radioman "A" School. In all four schools, the chart 
procedure^ resulted in shorter course_completion times when compared to compleyon 
times of control students in normally o^rated complexes without the. charts. Again, the 
student and instructor attitudes were strongly supportive of the effectiveness of the chart 
procedures. - , 

Conclusions . ^ / 

« * » 

Charted feedback^of progress in a CMI cour>5e is effective for improving student 
progress without interfering with achievem^t. / 

\ . ' * - 

Reco m m endat i ons _ . . 

The Chief of IJ^Javal Education and Training shoukJJl) implement the incentive chart 
procedures into CMI courses as a part of.any effort to Upgrade the.Cljiil system, and (2) 
investigate the fusibility of establishing a special service ribbon to be used as an 
incenjiye for superior training accomplishment. ^ 



J- 



FOREWORD 

This research was performed under work, unit ZlI76-PN.0i (Improving the Navy's 
Computer-managed Training System), as part of an R&D project aimed at improving the 
Navy's operational cx>m outer- manaj^d instruction (CMl) system, - It was sponsored by fhe 
Deputy Chief of Naval Operations (OP-01). 

This report is the last in a series" of five on Navy CML Previous reports have 
V described the problem areas that limit the effectiveness of the CMI system and the R&D 
plans that have been developed .to address theae problem areas (NPRDC SR SO-33), the 
effects of two student/instfuctor ratios on stucJent performance an<^ instructor behaVlof 
(NPRDC TR 81-6), the effects of automated performance testing on testing and training 
time at the Radioman "A" school (MPRDC TR 81-7), and the effects of test item format 
on learning and knowledge retention (NPRDC TR Si^ZU This report discusses the benefits 
from computer- generated feedback charts that display information on ctffnulative student 
progress as a motivation factor in CML Results of the CMI research will be used by the 
Chief of Naval Education and Training, the Chief of Naval Technical Training, comman- 
ding officers of all the Navy CMI schools, and -others concerned .With coiYi put er- based 
instruction. • ^ 
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INtRODUCTIChN 



Probtem 



One reason' why the Navy introduced corpputer management of instruction was to 
accommodate varying ratei of student progress throi^lv technical training courses. 
However, the ijidividualized instructicHi process associated with computer-managed in- 
struction (CMI) can be degraded by poor motiva^ti'on of^udents, *^ho are responsible for 
their own process, / Although both ne^tive and^ posmve Incentives are provided to 
encoqrage students to complete , courses , ahead of their predicted co^ipletion times, 
students are not always aware oi. their progress status.^.The CMI system needs a method 
of prowding progress feedback to students in technical schools to improve their progress 
through CMI course. . - ' ' \ , 

' Background . ' . 



problem of charting the course-related activities of students enrolled in self-^ 
>llege courses has received some attention (Hursh, 1978;'^Glick & -Semb, 1978). 



The , 

paced college wui^cd iitu> receiveu some axienuon vnursn, i:;/5;^Va**^,> x^/w 
Although students are* allowed to proceed at their individual pace in these courses, they 
are con^strained by academic icalendar units (qUftrters or senesters). Freed of externad 
checks, many students procrastinate until they fail to meet the completion date. 
Successful incentives to motivate students include bbrHis points^ for on-schedule comple- 
tion (Bitgood & Keech, 197li Bitgood & Segrave, 1975; Ll^, 19il), limiting the time that 
materials ape availabTe CHecWer, 1976), ahd progress charts (S^b, Conyers, Spencer, & 
Sanchezr-Sosa, 1975). An evaluation is needed of the potential effectiveness of progress 
charts in Navy training settings. ' 

^ 

Progress charts allow ^^udents to comp^e their daily actual progress rates with 
predicted rates (Figure 1 shows a predicted-rate plot). The Air Force, in a variation of 
this techhi<^e, uses manually- generated predictions to individualize suggested progress 
rates. The Air Force requires students to maintain their own charts and has developed a 
training module to provide ir^truction in this skill. McCombs, Dobrovolny, and Judd 
(1979) reported that these time-management procedures yield^ significantly shorter 
training times. ^ ' - ^ 



PrejJtcted Rate Line 



\ 




Course Days 



Figure 1. Pr^icted Progress Rate--sample progr>ess chart shoeing the 
predicted rate line (adapted from Semb etal., 1975). 



The Navy CMI computer routinely provides tffe data f rom which student progress can 
be predicted. A system of multiple-regression equations has been /leveloped tb^t relates , 
educ^ition history data and Armed Services Vocational Aptitude Battery (ASVAB)^cores to' 
completion times. By inserting the individual values of these variables into the equation, 
completion time for each terminal l^rning objective (TO) or instructi,onal module^>yithin 
the course can be predicted. The sum of these TO times is a predicted completion time 
(PCT), expressed in hours, for the entire CMI course. At any point in tbe course, a 
i learning rate (LR) pn be compute as the ratio of the time actually spent divided by the 
predicted time (actufil time t predicted time). This quotient indicates whether- this 
student is ahead of or behind schedule-- values greater than 1.0 indicate thei student is 
behind schedul^e; and-value? less than 1.0, ahead of schedule. To determine a student's 
progress rate tPR) toward the original predicted graduation day, the actual time is 
credited with time spent in extra study. ' 

The PR is sensitive to all of the student's efforts, to stay on schedule, because it 
considers as actual time. only the daily 6-hour bl&cks of time when the student is present 
in a CMI learning center. Since students can improve th6ir PR by study during off-hours, 
LRs, are always equal to or greater than PRs. The difference reflects the amount of extra 
effort the student expends to remain on schedule. These numerical ratios, along with 
other descriptive information, are provided daily to aid instructors in spotting students 
who are having problems and beginning to fall behind schedule. 

> 

Anecdotal evidence suggests that sharing these ratios with students, in conjunctipn \ 
with use of an informal incentk/e system, can reduce course completion time. However, 
many Navy instructors feel that, if students are shown the displays, they become overly 
sensitive to time factors to the detriment of learning. Despite_lhe reservations of some 
instructors, the Basic Electricity and Electronics (BE/E) School, Memphis, began printing 
predicted and actual times on the learning guide statements students receive after 
completing a test. Although the statements included predicted and actual TO completion 
times, they did not provide cumulative overall progress or the information necessary to 
compute such progress and did not list the criteria for incentive awards. 

Data covering the months immediately before and after this innovation showed no 
difference in completion times. These results, in conjmction with thc»e in available 
literature, suggest that PR data should be graphically displayed cumulatively, as in Figure 
* 2. This would allow students to forecast progress and see dearly how better study habits 
CQuld accelerate progress and ensure that they complete the course on time. ^ 

Objectives . ' - 

■9 

The objectives of this research were to (1) develop computer-generated feedback 
charts that display information on cumulative progress to motivate students to increase 
effort, (2) determine the best procedures for chart delivery, and (3) assess the feasibility 
of the chart procedures in operational CMI courses. 

APPROACH 

This investigation involved two exp^iments. The first experiment, conducted at one 
CMI school, was designed to determl^^experimentally the best method for delivering 
^ charted progress information. The second experiment, conducted at four CMI schools, 

was designed to assess the operational feasibility and generalizability of the best chart 
delivery method identified in Experiment I, across a range of course content and school 
management styles. 

• 
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CMI Course 



The BE/E School, Memphis, was selected for the investi^tion ol incentive charts to 
promote student f>rogreSs mder CML BE/E is ideal for^ an investigation of incentive 
variables for several .realms: ^ 

1. Since school aftendance occurs early in thl^ technical training series, stpdents 
include those whose inherently poor motivation to learn has not yet eliminated them from 
<the training system. 

2. Since BE/E is the largest training course, enrolling about ,25,000 students 
annually, an effective technique af BE/E would result in greater benefits- -reduced costs 
and training time--than could be obtained fro^ a course with few^r students^^ 

3. The chart procedures introduced in BE/E could be implemented iminediately in 
advanced cof r^es for which BE/E is a prerequisite, with no reintroduction being necessary/ 
If charts were to become a regular feature of CMI, students s^uld become familiar with 
thgm ip their first CMI course. This is the point in their training at which they leatrn the 
basic mechanics of interacting ^j^^ith the computer. 

The test sites were two feE/E learning centers, each housing 50,student carrels and a 
10-<;arrel central test section, thus, testing and l^rning occurred in the same phVsical 
Space.^ Each complex contained^^'^ optical scanner for entering test data and a General 
Electiric (GE) terminet for printing studeht learning guides. 

Materials 

The course materials were Modules 1-1^ of Coursefile (CF) 70 comprising 32 I^Os. 
Predicted completion times were available for' each TO. The student had to achieve 100 
percent mastery^ on tests for each TO before moving to the next. Remedial tests were 
assigned as necessary. Before graduating^ all students had, to pass a comprehensive 
examination with at least ^a 70 percent score. Time recfjired to take and pass i$he 
comprehensive examination was not included in training completion times. 

Chart Development i 

Three basic requirements influencled the progress display format (Johnston &. Penny- 
jxicker, 1980>. 

1. Progress data must accurately sl>ow how far the student is ahead of or behind 
schedule at any selected time. J 

2. The dis[day must be sufficiently sensitive so That^^small ^ins are immediately 
visibk|^tQ the student, v ' . * 

3. The data must be spread over enough time so that the student can project his 
progress andfif necessary, alter his study tactics to meet or exceed projected completion 
dates. ^ , 

In the first steps toward developing an effective chart, several variations oi the 
cumulative progress chart used by Semb et al. (1975) and th^ A'ir Force were tested on 
historical data from the Management Information and Ir^tructional Systems Activity. 

/■ . ■ .--3 9/ 
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(MIISA) /computer files. None of the chart versions test^ had both the necessary ra'hges 
and serfeitivity. . \ 

Navy instructors often treat the CMI daily .tabular progress report as a chart by 
tracing PR values as though tracing points on a graph. Accordingly, the new chart? were 
desired to display the PR values as a finction of TO rather than as a linear finction of 
time. Such a display was expeaed to be more acceptaWle to instructors and students 
because it resembled data already in use. The "coordinates' of the protted points were the 
TO that was just completed anjf the cumjilative progress rate to that point, defined as the 
sum oi the actual study hours minus extra study. hours divided by. predicted hours. The 
predicted value was displayed as a horizontal, line at the LO-level ordinate. Therefore, 
points below this line indicated that the student was ahead of schedule; and those above, 
behind schedule. • ' " 

The pi-ohlem remained of developing an. ordinate that would exhibit both sufficient 
range' and sensitivity. It was found that a two-cycle logarithmic scale \vith e as the base 
could accommodate all but a tiny fraction of observed cases with adequate sensitivity - 
, near the modal values around 1.0. Therrfore, the resulting chart' format, which is shown 
in Figure 2, was a two-cycle semi-logirithmic chart with PR.values from .37 to 2.72 on 
the ordinate, where the midpoint is 1.0 (the expected PR value), and^the TO numbers are 
on the abcissa (Figure 2). Also shown in Figure 2 are the incen.tives at designated ordinate • 
values. This incentive information was rtot -shown to ail students. However, when it -was 
used, it served as a constant reminder of the available Incentives -and the student's 
position in relation to them. The vertical line at TO 10 represents the point at which 
students be^n receiving and using the . 
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^ * . Figure 2. Studertt CMI progress chart showing available " 

I ^ incentives and hypothetical data. 
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Chart Delivery^ ?- ' ^ * . v * ; 

To prevent the res^rch activities fromMmpeding regCilar training operatioasl 
production of the charts had'to be autinnated, with^ delivery occurring at approximately 
the normal rate of studenf interactidq^ with the CMI system. Itlvas not practical to use 
theCMi central computer because of cost, ddays, agd interference with other instruction 
functions. Thus, an IBM 5110 system was programmed to maintain records djjplicating 
rthose in the central computer and to generate charts, complete with data points, of the 
type shown in Figure 2. ' ' • 

To ensure compatitility with^the larger system, the printing capai^biUtie^ of the IBM 
^5110 were limited to those of the GE terminets,and the satne paper was used. Each chart 
had 56 lines of print and was printed in 10 to 50 seconds, depending on the number of 
points to be computed. Input, filing, retrieval, computing, and plotting routines were ^ 
written in the APL computer lar^uage. 



Subjects 



Subjects were ^20 BE/E students (91 Navy and 29 M^irine Corps personnel). Subject' 
ratings feire shown iiV Table 1. j» 



■/ ■ - ■ Table 1 ' 

Ratings of Subjects for Experiment I 

^' . / % , • 
-■■ I ■ - .. * ^ 

Rating , - ■ • 

■ ■ , 1 . y 

Aviation Antisubmarine Warfare ' " . ^* 

Technician (AX) . / * , * " ^• 

Aviation Electrician's Mate (AE) , ^ ^ 

Aviation Electronics Technician (AT) ' ' . , ^3 * 

Aviation Fire Control Technician (AQ) ^ 6 

Aviation Support Equipment ^ 

' Technician (Electrical) (ASE) * 1 ^ 

Aviation Ordnanceman <A0) ' '5 

Avionics Repairman (AYR) • * -^13 

Tradevman (TD) ^ 8 ' 

Total 



120 



Seventy-one students in one complex were randomly assigned to ^on^ of four 
experimental groups, with each group receiving the charts the uncier different specified 
conditions; . , 



Group CC ( Constant Chart) (N ^ 16)^ tudents received a chart at the comple- 
tion of each TO, beginhing^with TC^IO. ' ' 

2* Group CS (Self-chart) (N = 21) -^At the end of TO 10, student^received a chart 
portraying their progress to that point.^They also received special ins^uctions (Appendix 



n 



A, A-4) on how to plot theic progress rate. Thereafter, only data coordii^tes nece^sary^ 
for the students to^, plot fheir own'progress were* provided at the completion Of each TO. 

.3. GroMP cp (Chart on Demand) (N = 21) -"Students received a cuijiulative progress 
chart at the end of XO ^O^hereSifter, charts were provided on student demand, i t 

' . . / * ^ - 
Group CY^(Chart Yoked to Dmand -(N = 13> --Studehts received accumulative ^ 
progress chart at the ^ of TO 10 and thereafter o^a r^indom baSis. Each CY student 
f waESsjna^ched with a studeat. Thws, each time the.CD student requested a chart, the 
computer automafically ^'elivefed'one to the CY student. The only difference between 
'the two groups was that the CD student requested the chart when he felt he need^ it, 
and the CY student received it re gardless ,of his perceived ne1fed.\eY yoking wa$viot 
initiated until several CD stu^dfitt^completed the course, so a complete history of CD 
chart requests was ayailable i^^H^Ht^^ yoking oi CY students. 



Near the end of diata coUecticxi, 11 additional students were randomly assigned tOk a 
fifth experimental group--Group CD^ (Chart on Demand 'with Incentive). This group 
followed the ^D. group procedure,' requesting charts as needfed. Howejver, the charts 
received on requesf had the incentive printed at the appropriate progress level (Fig*ure 2) 
to test the effect of charted feedback and incentive feminc^ ' . • . ^ 

The remaining 38 students in another complex constituted the control group. These 
students be^n the^cpurse during. the conduct of the experiment, co(npleted the course 
before data collection ended, and signed a release allowing their data to be used in the 
analysis. The control gro^p Was not| fof'mally informed of the research in progress,' 
although proximity to the site and assopation with the research subjects probably made 
them awace of the e^^rdse. ^ \ ^ / 

Pieced ute ' ' . ' ' 

^ — ' 

Otrin^ the first*dAf in the^learning center (LC), a student registered, received stu^y 
^hiaterials, and attended an introductory session' at which the LC instructor explained 
procedures and introduced the civilian, res^rch team m^ber. Volunteers for the study 
.were solicited (all but one student agreedvto participate) and were issued f/reliminary 
instructions (Appendix A, p. A-1). The computer generated ^tudent assignments for 
experimental conditions and created a data file that included the 32 predicted TO 
completion times obtained from MII5A. 

At the end of TO 10, the computer automatically printed a progress chart. 
Instructior^ appropriate to the assigned experimental condition (Appendix A, p. A-2) were 
given to the student, the chart was explained, with special emphasis orwifiterpr^tation of 
progress to date, and the course was continued. As indicated previously, students in 
Group CS received ^dditiortal instructions (Appendix A, p. A-^). ^ 



Re^rdless of the experimental condition to which a student was assigned, at the 
pmpletion of each TO, the^5110 computer printed the actial time required to complete 
t}iat TO and the time predicted for the next one.' This' predicted time was pirposely 
omitted from the learning guides to ensire tlmt t"he stuclfent jassed by the data collection 
site to obtain this information. At the end of the course, students took a comprehensive 
examination before being ass^ed to a f ollQW-on school or to a new duty station. - 

*At the end of the experim^rttj^ythe ^participating instructor^ and the 82 students in 
the experilnental^groups answered-a brief que&ticrifiaire designed to obtain their opinions 
on the value of the experijnental materials ana the procedures. 
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^ Data Analysis 

- The plan iot the data analysis consisted of determining the initial equivalency orjpthe 
five experimental groups and one control group by performing a one-*way analysis of 
variance (ANOVA) onjhe predicted overall completion times. .If there were no iigniiicant 
differences, the groups could be considered equivalent, and the effect of the charting 
procedures could be determined by performing a similar one-way ANOVA on the actual* 
course doolpletion times. 'If the groups were not equivalent, then the differences could' be 
contrqlled statistically by using an analysis of covariance (•ANCQVA) with the predicted 
corhpletion times being the covariate. ' - \ . - ' 

/ * <• 

R i esul'ts ■ . ' . . , 

i • 

Course Completion Times 

Table 2 presents ^the m^n predicted and actual completion tin^ies for the six groups. 
The one-\^y ANOVA performed on the predicted timp was not'' significant (F = .523, 
df =5, ll*j p > .05). This indicates that the groups are equivalent and confirms the use of 
the ANOVA on t^e actual completion times to determine the effects of the chart 
procedures. - Here a^in, the ANOVA revealed no significant differences among the six 
grou[^ (F = .737, df = 5, 11*). In case the effect of , the chans was maskjsd due to 
differential ix-edicted completion times that .were not detected by the ANOVA, an 
ANCOVA was performed using the predicted times as the covariate. and^ the actuaJ^mes 
as ^he dependent measi^e. Tfjis" analysis also revealed no si^if leant diffeWnces 
(F = l.*25, df = 5, 11*). While* this overall iinding did not pinpoint a statistically superior 
^chart procedure, comparing irBividual chart groups means with the mean for the control 
group, along with other factors, suggested the need for further chart procedure evalua- 
tion. For example, the combined m^n for the two demand groups (CD and GDI) was 92.0 
hours, 10.7 hours less than the mean for the control group— well over 1 full training day. 
.Since the tOI group required less completiron time than did the other groups, the chart on 
demanrf^^iW incentive indicatofs procedure was selected for use in Experiment II, Also, 
this procedure was positively supposed by students and instructors and promised, to 
reduce training time significantly* 

\% , * ' 

Tabl-e 2 

^ ^ ^ Group Mean Predicted and Actual Completion 

Times and Test Scores ^ ^ f 



Group 


M^n Predicted 
Completion Jime 
(Hours) 


Mean Actual 
Comple^on Time 
(rfours) 


Mean Comp. / 
Score 


Constant Chart (CC)(N= 16) 


10.2.6 • 


# 98,5 


79,81 


Self Chaft (CS) (N = 21) 


106.2 


105,3 


'79.7k 


Chart on Demand (CD)*(N = t\) 




9«t.7 


81.67 


Chart Yoked. to Demand fcY) (N = 


13) 98.1 , 


96.9 


77.80 • 


Chart on Denftand with Incentive 
' (CDI)(N= U) 


105.3/ 


87.2 


82,50 


CQntrol(NV3g) 


101 .2 


102.7 


' 81.00 
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Comprehensive Test Scores * / 

To assess the effect of the trhart procedures on overall course achievement, an 
ANQVA was performe^I on the final comprehensive fest scores,, which are alsck, provided In 
TaUe 2. The ANOVA faults indicated no significant differences in mean:j^fes across 
th^ chart and cbntrol groups (F = 0*66*, df = 5, In, other wo<-ds, as a group, the 

students. performed equally well in all conditions on the corrip^iehensive end-of-course test. 

Attitudes ' . ^ l \ 

Attitudes about the chart procedures , were assessed by means of a brief question- 
naire. Since many of the students were reassigned imrr^iately after finishing the course, 
, completed questionnaires were available from only 6Fof the 82 experimental students. 
The overwhelming majority of tfee responder^ts indicated that they liked and understood 
tKe progress charts. Stiofents also indicated that the charts would be a good way for 
instructors to follow studfent progress and that the charts should be placed in all QMI 
courses. Conversations with instructors confirmed this idea: They indicated the desire to 
have the charts avaitable to them and to have additional information such as student 
progress data on an individual module basis. 

. , In a final questionnaire item, students were asked to. dtvide 100 points bet-ween 
various incentives in an effort to determine the rdative values of incentives that ,could be 
^ offered. As shown in Table 3, students assigned the highest rating to the letter of 
commendation. The frivolous incentive that was included— free beer and dancing- — 
received only 3.2 points, whidi suggests how keriously students view the incentives. The 
predominant entry in the "otnei^' category w^ the satisfaction of having ilf^stered the 
course cont^t. For some students, tan^ble incentives are not necessary in a^ genuine 
learning experience. 



Table 3 

Incentives and Mean Assigned Point Values 



' § \ \ ^ 

Incentive ^ Mean Assigned Point Values 



Letter of Commendation* , ^ 


30.5 


Day of Liberty 


19.5 


Ribbon for Acadenrfic Distinction » 


. . -17.7 


Letter of Appreciation 


, 13.2 


Other . 


\ ' 8.1 


Special Graduation Certificate 


8.0 


Free Beer and Dancing 


3.2 


Total' 


100.2 



/ 
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. The suggested incentive of a special decorative ribbon for academic achievement was 
assigned 17.7 points, compared to 13.2 ^or an existing incentive, a letter of appreciation, 
and 19.5 for a day of liberty. Since service ribbons clearly are seen as valued ^rms of 
visible recognition, they should be considered when contemplating additions to^raining 
incentives; ' , " . * 

, . Although instructor enthusiasm was somewhat more restrained than ^t of the 
studentts, seven out of eight instructors endorsed the proposal to add charts to the system. 
Six qi the eight instrOctors^ favored having ready access to, progress data in chart form. 
As the study progressed, som? instructors became quite skilled at using ^he charts as an 
* aid when advising on studystactics and special problem areas, > 

V 

♦ EXPERIMENT 

. As indicated previously. Experiment II was. conducted to cletermine how well the 
chart procedure%work in an operational CMI getting. This experiment was conducted in 
four CMI schools, each with its own subject matter and management style, using the Navy 
CMI computer, and with no civilian researchers present. 

CMI Courses ' ^ , , 

— ; ' 

CMI courses selected for this experiment were those conducted at the BE/E Schools, * 
Great Lakes and Memphis, the Pr(^ulsion Engineering (PE) Basils School, Great Lak^, 
and the Radioman "A" School, San. Diego. These courses, differed in content and^ 
management style but permitted a relatively low-cost research effort through some^ 
common locations. A fifth school was to be used in the experiment but time limitations 
prevented collecting^ adequate data from that school. The charting procedure used was 
the chart on demand with incentive. Although the same chart procedures were 
implemented in ail four schools, pfiysifcal constraints and management prerogatives did 
ififluence the precise manner in which the procedures were carried out. These differences 
will be discussed in the section on chart delivery. 

For each school, the test site involved all, students in an entire learning complex. A 
complex is typically comprised of from three to five learning centers (LC), with each LC 
being made up ol 20 to^30 students managed by a learning center irstructor (LCI). In 
some complexes, study and testing occurred in the same location. In other comple;ces, a 
separate LC was 'designated just for'testing. Each complex had the normal allotment of 
one optical scanner and one GE terminet for regular CMI system interactior^. In addition, , 
for this experiment, an extra GE terminet was fx-ovided for the generation of the 
incentive charts. The CMI system computer software had been specially modified by ^ 
MIISA to permit chart* production at the extra termi^ets with complete up-to-date student 
progress information. ^ " , ^ * 

Materials 

/• 

The course materials included the. textbooks, manuals, and equipment normally used 
with each course. The only special materials for this experiment ^vere a sign designating 
the extra terminet for "Chart Use Only," a one-page student instruction form describing 
the charts and how to get them, two pages of instructions about the charts for the 
instructors, and the charts themselves. 

' ^ • / ' 

. Chart Development 



^ The chart developed for student^ ise evolved from the one used in Experiment I, with 
modifications to suit the operating requirements of the conventional CMI equipment. 



Figure 3, which provides a sample^of this student currvulative progress rate chart, shows 
that it includes four incentive indicating lines: (1) review for Academic Review Board 
(ARB), (2) deficient progress- -mandatory quiet study (MQ5) recommended, (3) excellent 
J>rogress, and (4) outstanding pnogr^ss. While each school could choose where on the chart 
ordfnate to locate these inceptive lines, they were generally located close to the following 
points, respectively: 1.50, 1 •25,' 0.80, and 0,50. 
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Figure 3. Studer^cumulative progress rate chart. 

From interactions wit|4 JnstruCtpH during Experiment I, it was learned that a 
different [and of chart for instruct6rs might also prove beneficial. The instructors not 
only wanted to know about the studenfs cumulative progress but also about his progress 
on each Individual module.* This information was useful when diagnosing student problems. 
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Accordingly, an 4ncli.vidiitl module progress chart for instructors was developed and is 
shown in Figure From this chart;, an instructor can see how/the student p^ormed on 
^ch module ir> terms of actual progress compared to predicted progress for that module. 
Two versions of the insfryctqr chart, were a'vailal)le. One version presented .data onlyTbr 
the last 20 modules completed,' and the other version included a printout of all course data 
for that student. 

r ' ^ ' . * ' . - ' \ , 
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Figure ^. Individual module Drogr,ess, chart. , Note:' Individual module 
progress rates (PRs) are indicated by an asterisk. The last six 
data points of a cumulative PR Une are shown by plus (+) 
signs. yt^ 

Chart Delivery ^ ^ \ - ' \ 

For all coyrses, the charts werfe delivered from the CMI system by means of ihe extra 
terminet located in the complex. In the ideal delivery situation, the students ^ould 
request charts by manipulating the^terminets themselves; however, Ihis procedure was not 
aUowed at the BE/E School, Great Lakes. Consequently, at that school, the student 
requested the ch&rffrom the instructor, who then obtained the appropriate chart from the 
terminet. An additional difficulty encountered at the BE/E SchSol, Great Lakes, was that 
an extra terminet was not available. As a result, the instructor had to perform additional 
entries into the regular terminet in order to obtain the chart. This requirement certainly 
ihcreased the difficulty -of getting charts and probably reduced the number of charts 
requested, by the students. If these procedures Were qonsidered for system-wide 
impiementation, additional computer software modifications could4)e made to eliminate 
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''the need for the extra terminets and to simplify- the method for requesting an:hart from^ 
the computer* s ■ * 

Subjects - - , 

Subjects, for this experiment included all students who were randoml^ssigned to the 
^ designated chaj* complexes after a specified date and who completed the Course during a 
3-Tnonth period. For all schools, this provided 568 chart students^ This timii^ method for 
subject selection meant that initially there were students in each, complex who were not 
. participating in the study. Brief observation and discussions with instructori.re^^aled 
that this p<^ed no problem. • 

Control subjects indud?d_aH 657 students who were assigned to different complexes 
that operatfgi without the charts under normal school procedures during the same time 
period. Paty were obtained for the control group students from the CM! system. The 
.^control complexes selected were as similar as possible to the chart complexes, exfcept for 
the use of the charts. - ' / 

Sample sizes for chart apd control groups at each school are provided in Table ^. 



^ Table * . / 

Sample Size for Ctert and Cdntrol Grouf^ 

' Experimer^t 11 ^ 



/ 


\ ' Number of Students 


School 


Chart Groups ^ 


Control Groups 


BE/E, Great Lakes * 


58 


99 


^E/E, Memphis 


' 268 


PE School , 


191 ■ 


176 


Radioman "A" 


/ 120 


11* 



Procedure . 

< 

Operational CM! procedCires'were used in the l^rning complexes as much as possible 
during this experiment. Researchers rarely entered the chart complexes and then only to 
interact with .the Navy instructors to verify-the use of proper procedures. IrstructoTs for 
the chart con^plexes were volunteers who received ^bout 2 hours^of instruction on the use 

* of the charts. After the start of the investi^tion, the instructor told each student who 
registered in the CMI complex about the charts and gave him or her the one-page student 
chart irstruction sheet.; The students were permitted to obtain their first cterts after 
they had completed the second module in the course. They could obtain new charts only 

^,after completing anotW imtructiona^ module and then only upon request. Instructor 
charts were available onl/ to. the instructors and^at their discretion. At the end of the^ 
data collection period, three schools, all but Propulsion E'n^neering Basics, continued 
i^ng the chart procedures until, computer system problems necessitated termination of 

^ chart generation. 
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Data Anal^ s 

r 

The data analysis plan for Experiment II duplicated that of Experiment I. For each 
school, the mean predicted course completion times were compared for the chart and 
/control groups^ If the groups were determined to be statistically equivalent, the effect of 
the charts was assessed by comparing the mean actual course completion times. Since 
each school' only had two groups, and each school had its own irxiependent course schedule, 
separate t>.tests were used. 

Results 



Coiyse Completipn'Times , , 

5, which presents the mean predicted completion hours "for chart and control 
group^f^ong with t-test results, shows that^ there were no significant differences 
between chart and control group. predicted times for any school. This outcome indicates 
that the two groups were equivalent in each school and that the actual completion times 
could also be analyzed using the t-test. These results, also provided in Table 5, reveal 
that chart students in aUVour schools completed actual training in a numericaUy shorter 
time than control students, with the difference being significant for two schools. The 
average difference between chart and control actual completion fimes was over 14 hours, 
whidi is more tten 2 CMI training days. , 



Table 5 



M?an' Predicted and Actual Course Completion Times for 
• Chart and Control Groups— Experiment II 



School 


Mein Completion Time (Hrs) 
Chart (N) Control (N) 


•M^n Difference 
- (Control-Chart) 


t-test 
Result 


^ Predicted 


BE/E Great L^kes 
BE/E, Manphis 
Propulsion Engineering 
Radioman "A" 


205 
150 
109 
209 


(58) 207 
/(199) 152 
(191) 113 

,n20)/ 205 ♦ 
^ 


(99) 
(268) 
(176) 
(114) 


2 

« 

2 

% 

-4 , 


*V0.21' 
jr.79 
1.47. 
0.99 


«' ' -•• 




Completed 








BE/E, Great Lates 
BE/E, Memphis % 
Propulsion Engineering 
Radioman "A" 


172 
128 
95 
196 


(58) - 195 
(159) 137 
(191) m' 
(120) 214 


< (99) 
(268) 
(176> 
(114) 


23 
9 
9 

18 


• 1.96 
2.01* 
2.55* 
1.84 



*P < .05. 
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Attitude^* , ^ . ^ 

^* / * • 

Eighty percent (23) of *t+ie 29 instructors responding to the questionnaire felt that the 
charts were useful in belj>in| the students complete the course efficiently, ' Over 72 
percent indicated that they would like to, have the charts, available to them when they 
manage anotherXMf course. Fifty-seven percent of the instructors and SO percent of the 
jstudents agreed that the students would volunteer for extra study to keepjtheir PR down, 

DISCUSSION AND CONCLUSIONS 

,Although course completion ^ata alone from Experiment I do not support the effiracy 
of the chart i>rocedures, other factors entered into the decision to conduct the operational 
evaluation in Experiment II. The strong support from both students and instructors tor the 
chdrt procedures and the sizable potential savings in reduced training time if the chart 
procedures were effective were sufficient rearsons to^proceed with the operational test in 
Experiipertt II. The best chart method appeared to involve students requesting charts that 
have Indications of 'available inc'entives. ^ 

Experiment 11 amounts tb four independent replications of the chart experiment, 
using as a chart procedure the Chart on Demand with Incentive coi^ition. In all cases, 
with four different schools and management st^es, fhe chart procedures did yield reduced 
training time* The procedures were of sufficient benefit that, aftei-^termination 6i data 
collection, three schools continued to use the charts. It is not possible from Experiment 11 
to tell whether the reduced training time was a resist of the student cl;iart^'the instructor 
charts, or all charts in combination. It is dear, however, that the chart procedures bring 
both the students and instructors into closer contact with the actual progress of the 
student so that training time reductiais occur'. Apparently, the students are made mor^ 
aware of their \mb of training time and consequently spend the time m^r^ judiciously." 
Re^rdless, the rfet effect In this case was to reduce training time on an average of 1 to 3 
training days. This Ending tepliqates that foiffKi by McComte, Dobrovolny, a*nd 3udd 
(1979) in an Air Fora CMI system and ext^s the generality of the cha'feting procedures 
to another instructional setting. 

4 

For this operational test, it was necessary to ir^tall an additional GE terminet ifl 
each complex in order to generate the charts. This would not be operationally practical. 
MllSA personnel have advised that computer Software modifications could eliminate the 
need for the extra terminets. In view of the need for software modification, argr Chart 
procedure implementation decision should be made in conjinction with decisions fe^rding 
any other planned upgrade of the CMI system. 



RECOMMENDATIONS 

It is recommended that the Chief of Naval Education and training (D'implement the 
incentive chart procedures into CMI purses as a part of any effort to upgrade the CMI. 
system, and (2) investigate the feasibility of establishing a special servic^ ribbon to be 
•used as an incentive for si4>erior. training accomplishmwt. 
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PREUMINARY INSTRUCTIONS TO STUDENTS 

Welcome to, Basic Electronics and Electricity! As you know, you are beginning the 
course that is fundamental to many of the advanced draining opportunities that probably^ 
attracted you to the Navy in the first place. You are certainly aw^re of how important, it 
is to you that you do well in this course. ^ . . 

Basic Electronics ^d Electricity is one of the courses taught^ by the Navy's 
c<5rhputer*managed instruction (CMI) system. The Navy's CMI system is the biggest, and ^ 
probably the best, in the world today. Thousands of successful graduates of BE/E report 
that they especially like the individualized aspect of the CMI .system. . 

>. No system is perfect, not even the Navy CMI system. For that reason, the Navy 
Personnel- Research and Development Center in San Diego has had a team of researchers 
studying the CMI system for several fears. We are part of that team and we are working 
in the center you are^ assigned to. We are conducting a study 4o see if certain kinds of 
feedback help students learn the material better and/or faster. We would like very mpch 
for you to participate in our study, "particularly since almost everyone else in the center 
will be participating. We can assure you that w^ will not ask you to do anything that 
would impede your ^performance in BE/E; in fact, we thihk you will be helped by 
participating. Here Is $11 you will have to 'do to participate: 

^ * / 

1. Read and iign the accompanying form. In order for us to use your data for ' 
research pucposes, we must have your signed permission. Your privacy will be completely 
protected! Our little computer is no smarter than the Navy's big computer and it, really 
■dQ§|nt care. who you are, only how well you are' doing. ' . ' 

« 

vGive the signed form to one of us so Ave can^ej^er you into our little, computefr. 
We'll be eisy to fif^d— sitting at the front af the centerfnear the terminet. 

3. Each time you have a transaction iwith the CMI computer and receive a message 
from the terminet, bring it by our table so one of us cdn enter the data we need into our 
little computer. We will immediately return your message to you^so you can file it with 
the others. This step is very important; to help you remember, we will have at our table 
th.e hole purfch you need ta put your transaction slip into your file. Just find the hole 
punch youMl find us! 

That's all you have to do for now—sign the form, sign up, and remember to stop by 
our Jable each time you go to thp terminet. y^fter you have completed about ten modules 
or so, we » will give you, some additional instructions about the feedback you will be 
receiving during the remainder ot the course. If^ou have an^ questions at all about this, 
one of gs will be happy to .try to S^^wer them ior you. ; 

Remember, your participation is important! Help us try to make BE/E ever> better 
than ^1 is now. 

NAVPEJISRANDCEN Research Team 

^ Bill Hartman 
' Linda'Ward 
Brian Brett 4 
Hank Pennypacker 
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PHASE E ir^TRUCnONS TO STUDENTS 

Congratulations! You are now between one quarter and one third of the way through 
BEff^. You are no doubt completely familiar with how the CMl'system works and how we 
are gathering data* From ^ere on, as we promised in the preliminary instructions; we will 
be providing some useful (we hope? information to you. * ' 

You have^just received a Progress Evaluation Chart (PEC) from our littl^e computer. 
This chart gives you a picture of your progress in BE/E as of this date. Here is what you 
need to know to interpret your chart: 

|. .The horizontal axis represents the 32 BE/E computer module tn the order you 
will take them. ' . a, • C ' 

• ' V 

2. The vertical axis is called Progress Rate, H^e is how it is calculated. Based on 
the experience of hundred^ of other students who have' taken BE/E, the Navy computer % 
makes a prediction as to how long you s)x)uid takd to complete each rftqdule. Of course it 
also knows how long you actually tate to complete each module, so it can tell whether you 
are ahead or behind of where it thinks you should be. It makes this calculation each tin(e « 
you begin and finish a module. i 

3. Each little symbol on your chart shows how your actual progress compares to 
your predicted progress up to that point in the course . The, Progress Rate is the total 
actual time divided by the total predicted time, so if you are ahead -of schedule, your 
symbols will be below the 1.0 line. }X you are behind schedule, yqur symbol will be aibve 
the 1.0 line. For example, suppose that your symbol for ModuM 6is-atH:;26^. This would 
mean that, after completing % modules, you had used 20 percent more time than the 
computer predicted you would. But, sq^pose that after Module IC, your symbol is down to 
0.90. That means Vou picked up speed on Modules 9 and 10 so that you are now 10 percent 
ahead of schedule. ^ ' 

The reason we are giving you these charts is to let you monitor your own progress, 
this will help you avoid unpleasant things like assigned extra study time or having to talk 
to an Academic Board. Even better, the Navy has some special rewards for people who^o 
much better than predicted, and we think progress chart? may help more students earn 
those bonuses. More about that later. For now, you should try to make your symbols go 
^^ISSi ^ possible on the chart. The way to do this is to work rapidly but carefully on 
each module so you will beat yoir predicted time. Don't work too fast ar»d get sloppy, of 
you will lose more time than you gain by having to take too many remedial tests. 

In order to help you keep track of your progress, we will be giving you: 

• An up-to-date chart lik^this every time you complete a module (Group CC). 

• The information you need to plot your own Progress Rate on the chart (Group 
CS). 

• An up-to-date chart like this at various times throughout the remainder of the 
course (GrQup CY).' 

• An up-to-date chart like this WHENEVER YOU ASK US FOR ONE (Group CD). 
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In addition, jj^e wi|l be giving you your predicted times for ifach module so you will be 
I ' better able to plan, your work time and bring your chart doWpf ; 

Yoy may notice that other students in the ctenter are receiving feedback more or less 
often than you. ThM is because we are trying to determine whether these types of 
feedback help students and if so, under what conditions they helf the most. To do this, we 
h^ve to have sbm'e variation in the schedule and type oi feedback given. 

If you have any questions at all', or need help interpreting your, chart,^ just ask any 'of 
us. Don't forget to keep coming by our tatole each time you have a transaction at the 
terminjt so we can keep our little computer up to date and give you accurate feedback! 

. ' NAVPERSRANDCEN Research Team 

Bin Hartman . 

i • , , Linda Ward — 

. , Brian Brett 

Hank Pennypacker 
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PROGRESS RATE PLOTTING GUIDE 

In orde^W help. you keep track of your progress, we are giving yoii y^ur current 
, progress rate every time you complete jnodyle. .You should plot this nu^noer on your 
Pwgress Evahation Chart .(PEC) in order to see how your progress hJs changed sinqe last 
time. prapWog on the PEC is ea^y and quick. Just follow these steps: ^ , 

^ 1. On the horizontal axis, read across until you find the module number you have 
completed. v . . ^ 

. 2. In the same manner, locate your current progress rate on the vettipal axis. 

3. From the module number located in Step 1, moye perpendicularly up the chart to 
a poif^t directly across from the value on the vertical axis ths^t corresponds to 
your current progress rate. . 

Place a dot on the spot located In Step 3. The dot, should be*directly over the 
module number you just completed and straight across from the progress, rate 
value equal to your current progress rate. 

Be sure to graph your new progress rette As soon as you complete a module so you will 
have, a complete up*to-date picture of your progress ^roi^h the course. 



